Electronic device and method for reliability-based object recognition

ABSTRACT

According to embodiments, an electronic device comprising a processor configured to receive an image including one or more objects, acquire a first one or more of the received one or more objects, acquire one or more reliability measures associated with the acquired one or more first objects, receive an input including information having one or more words, acquire one or more second objects corresponding to at least a part of the one or more words, when there is at least one object corresponding to the one or more second objects among the one or more first object, adjust at least one reliability measure corresponding to the at least one object among the acquired one or more reliability measures, and recognize the one or more first objects by using an image recognition scheme at least based on the one or more reliabilities including the adjusted at least one reliability.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2017-0146639, filed on Nov. 6, 2017,in the Korean Intellectual Property Office, the disclosure of which isincorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure relates to image processing method and device.More particularly, various embodiments of the present disclosure relateto an electronic device configured for improving an object recognitionrate, and to an operation thereof.

BACKGROUND

Electronic devices are capable of identifying objects in images. Uponidentifying an object in an image, the electronic device can enhance theappearance of the object. However, in view of the foregoing, it isimportant to have a high object recognition rate.

SUMMARY

Electronic devices are capable of processing an image by acquiring a rawimage through an image sensor and processing the acquired raw imagethrough an image signal processor (ISP). The ISP can process the rawimage by using an image quality improvement algorithm, thus providing animage with improved quality. The ISP may perform various kinds ofprocessing such as white balance adjustment, color adjustment (e.g.,color matrix, color correction, color enhancement), color filter arrayinterpolation, noise reduction processing or sharpening, or imageenhancement (e.g., high-dynamic-range (HDR), face detection). An imageoutputted from the ISP may have, for example, a YUV format. Also, animage outputted from the ISP may be compressed, for example, using JPEGmethod, and the compressed image may be stored in an electronic device.

Meanwhile, an image processing cloud system is being used for imagebackup and new media content creation. In this system, an image may beuploaded to a cloud server, and a computer vision-based technique suchas image matching that is difficult to perform in a client device may beapplied to the uploaded image. Using machine learning based software,for example, the cloud server can perform image recognition.

An electronic device can recognize an object in an image by using arecognition algorithm acquired through machine learning or deeplearning. Also, the electronic device can perform image processingthrough such object recognition and provide the processed image to theuser. However, an error in object recognition may cause inaccuracy ofimage processing, for example, inaccurate editing of an object unlike auser's desire.

Various embodiments of the present disclosure provide an electronicdevice configured to improve the reliability of accurate objectrecognition and also provide a method for related operations.

According to various embodiments of the present disclosure, anelectronic device comprising a processor configured to receive an imageincluding one or more objects, acquire a first one or more of thereceived one or more objects, acquire one or more reliability measuresassociated with the acquired one or more first objects, receive an inputincluding information having one or more words, acquire one or moresecond objects corresponding to at least a part of the one or morewords, when there is at least one object corresponding to the one ormore second objects among the one or more first object, adjust at leastone reliability measure corresponding to the at least one object amongthe acquired one or more reliability measures, and recognize the one ormore first objects by using an image recognition scheme at least basedon the one or more reliabilities including the adjusted at least onereliability.

In another embodiment, there is presented an electronic devicecomprising a touch-sensitive display; an input device; a camera; acommunication module; and a processor functionally connected to thetouch-sensitive display, the input device, the camera, and thecommunication module, wherein the processor is configured to: acquire afirst image through the camera, display a second image corresponding tothe first image on the touch-sensitive display, acquire a user inputthrough at least one of the touch-sensitive display or the input devicewhile the second image is displayed, transmit the first image and theuser input to the external electronic device through the communicationmodule such that the external electronic device performs recognition ofan object corresponding to the user input among objects of the firstimage, receive a result of the recognition from the external electronicdevice, acquire a first user input responsive to the result through atleast one of the touch-sensitive display or the input device, andtransmit the first user input to the external electronic device throughthe communication module such that the external electronic deviceadjusts a reliability associated with the object recognition, based onthe first user input.

A method of operating an electronic device, the method comprisingidentifying an image including one or more objects; acquiring one ormore first objects corresponding to at least a part of the one or moreobjects, and one or more reliabilities associated with recognition ofthe one or more first objects; acquiring an input including informationhaving one or more words associated with the image; acquiring one ormore second objects corresponding to at least a part of the one or morewords; when there is at least one object corresponding to the one ormore second objects among the one or more first objects, adjust at leastone reliability corresponding to the at least one object among the oneor more reliabilities; and recognize the one or more first objects byusing an image recognition scheme at least based on the one or morereliabilities including the adjusted at least one reliability.

According to various embodiments of the present disclosure, theelectronic device can improve the reliability of object recognition byperforming the object recognition based on a user input.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an electronic device in a networkenvironment according to various embodiments.

FIG. 2 is a block diagram illustrating a camera module according tovarious embodiments.

FIG. 3 is a conceptual diagram illustrating operations of an electronicdevice and an external electronic device according to variousembodiments.

FIG. 4 is a block diagram illustrating an electronic device and anexternal electronic device according to various embodiments.

FIG. 5 is a flow diagram illustrating operations of a processoraccording to various embodiments.

FIG. 6 is a flow diagram illustrating operations of a processor forimproving an image recognition rate according to various embodiments.

FIG. 7 is a diagram illustrating operations of an electronic device andan external electronic device according to various embodiments.

DETAILED DESCRIPTION

In certain embodiments, a scheme for improving object recognition inimages is presented. The image is captured by a camera and includescertain objects. The objects in the image are identified using an imagerecognition algorithm. The identified objects are also associated withreliability scores, wherein a higher reliability score indicates ahigher confidence that the identified object, is in fact, an object.This is a first group of objects. A second group of objects can beacquired from a user input (e.g., a user's utterance) while displayingthe image. When any object in the first group exists in the secondgroup, the reliability for the object is improved.

FIG. 1 is a block diagram illustrating an electronic device 101 in anetwork environment 100 according to various embodiments. Referring toFIG. 1, the electronic device 101 in the network environment 100 maycommunicate with an electronic device 102 via a first network 198 (e.g.,a short-range wireless communication network), or an electronic device104 or a server 108 via a second network 199 (e.g., a long-rangewireless communication network). According to an embodiment, theelectronic device 101 may communicate with the electronic device 104 viathe server 108. According to an embodiment, the electronic device 101may include a processor 120, memory 130, an input device 150, a soundoutput device 155, a display device 160, an audio module 170, a sensormodule 176, an interface 177, a haptic module 179, a camera module 180,a power management module 188, a battery 189, a communication module190, a subscriber identification module (SIM) 196, or an antenna module197. In some embodiments, at least one (e.g., the display device 160 orthe camera module 180) of the components may be omitted from theelectronic device 101, or one or more other components may be added inthe electronic device 101. In some embodiments, some of the componentsmay be implemented as single integrated circuitry. For example, thesensor module 176 (e.g., a fingerprint sensor, an iris sensor, or anilluminance sensor) may be implemented as embedded in the display device160 (e.g., a display).

The processor 120 may execute, for example, software (e.g., a program140) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 101 coupled with theprocessor 120, and may perform various data processing or computation.According to one embodiment, as at least part of the data processing orcomputation, the processor 120 may load a command or data received fromanother component (e.g., the sensor module 176 or the communicationmodule 190) in volatile memory 132, process the command or the datastored in the volatile memory 132, and store resulting data innon-volatile memory 134. According to an embodiment, the processor 120may include a main processor 121 (e.g., a central processing unit (CPU)or an application processor (AP)), and an auxiliary processor 123 (e.g.,a graphics processing unit (GPU), an image signal processor (ISP), asensor hub processor, or a communication processor (CP)) that isoperable independently from, or in conjunction with, the main processor121. Additionally or alternatively, the auxiliary processor 123 may beadapted to consume less power than the main processor 121, or to bespecific to a specified function. The auxiliary processor 123 may beimplemented as separate from, or as part of the main processor 121.

The auxiliary processor 123 may control at least some of functions orstates related to at least one component (e.g., the display device 160,the sensor module 176, or the communication module 190) among thecomponents of the electronic device 101, instead of the main processor121 while the main processor 121 is in an inactive (e.g., sleep) state,or together with the main processor 121 while the main processor 121 isin an active state (e.g., executing an application). According to anembodiment, the auxiliary processor 123 (e.g., an image signal processoror a communication processor) may be implemented as part of anothercomponent (e.g., the camera module 180 or the communication module 190)functionally related to the auxiliary processor 123.

The memory 130 may store various data used by at least one component(e.g., the processor 120 or the sensor module 176) of the electronicdevice 101. The various data may include, for example, software (e.g.,the program 140) and input data or output data for a command relatedthereto. The memory 130 may include the volatile memory 132 or thenon-volatile memory 134.

The program 140 may be stored in the memory 130 as software, and mayinclude, for example, an operating system (OS) 142, middleware 144, oran application 146.

The input device 150 may receive a command or data to be used by othercomponent (e.g., the processor 120) of the electronic device 101, fromthe outside (e.g., a user) of the electronic device 101. The inputdevice 150 may include, for example, a microphone, a mouse, or akeyboard.

The sound output device 155 may output sound signals to the outside ofthe electronic device 101. The sound output device 155 may include, forexample, a speaker or a receiver. The speaker may be used for generalpurposes, such as playing multimedia or playing record, and the receivermay be used for an incoming calls. According to an embodiment, thereceiver may be implemented as separate from, or as part of the speaker.

The display device 160 may visually provide information to the outside(e.g., a user) of the electronic device 101. The display device 160 mayinclude, for example, a display, a hologram device, or a projector andcontrol circuitry to control a corresponding one of the display,hologram device, and projector. According to an embodiment, the displaydevice 160 may include touch circuitry adapted to detect a touch, orsensor circuitry (e.g., a pressure sensor) adapted to measure theintensity of force incurred by the touch.

The audio module 170 may convert a sound into an electrical signal andvice versa. According to an embodiment, the audio module 170 may obtainthe sound via the input device 150, or output the sound via the soundoutput device 155 or a headphone of an external electronic device (e.g.,an electronic device 102) directly (e.g., wiredly) or wirelessly coupledwith the electronic device 101.

The sensor module 176 may detect an operational state (e.g., power ortemperature) of the electronic device 101 or an environmental state(e.g., a state of a user) external to the electronic device 101, andthen generate an electrical signal or data value corresponding to thedetected state. According to an embodiment, the sensor module 176 mayinclude, for example, a gesture sensor, a gyro sensor, an atmosphericpressure sensor, a magnetic sensor, an acceleration sensor, a gripsensor, a proximity sensor, a color sensor, an infrared (IR) sensor, abiometric sensor, a temperature sensor, a humidity sensor, or anilluminance sensor.

The interface 177 may support one or more specified protocols to be usedfor the electronic device 101 to be coupled with the external electronicdevice (e.g., the electronic device 102) directly (e.g., wiredly) orwirelessly. According to an embodiment, the interface 177 may include,for example, a high definition multimedia interface (HDMI), a universalserial bus (USB) interface, a secure digital (SD) card interface, or anaudio interface.

A connecting terminal 178 may include a connector via which theelectronic device 101 may be physically connected with the externalelectronic device (e.g., the electronic device 102). According to anembodiment, the connecting terminal 178 may include, for example, a HDMIconnector, a USB connector, a SD card connector, or an audio connector(e.g., a headphone connector),

The haptic module 179 may convert an electrical signal into a mechanicalstimulus (e.g., a vibration or a movement) or electrical stimulus whichmay be recognized by a user via his tactile sensation or kinestheticsensation. According to an embodiment, the haptic module 179 mayinclude, for example, a motor, a piezoelectric element, or an electricstimulator.

The camera module 180 may capture a still image or moving images.According to an embodiment, the camera module 180 may include one ormore lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to theelectronic device 101. According to one embodiment, the power managementmodule 188 may be implemented as at least part of, for example, a powermanagement integrated circuit (PMIC).

The battery 189 may supply power to at least one component of theelectronic device 101. According to an embodiment, the battery 189 mayinclude, for example, a primary cell which is not rechargeable, asecondary cell which is rechargeable, or a fuel cell.

The communication module 190 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 101 and the external electronic device (e.g., theelectronic device 102, the electronic device 104, or the server 108) andperforming communication via the established communication channel. Thecommunication module 190 may include one or more communicationprocessors that are operable independently from the processor 120 (e.g.,the application processor (AP)) and supports a direct (e.g., wired)communication or a wireless communication. According to an embodiment,the communication module 190 may include a wireless communication module192 (e.g., a cellular communication module, a short-range wirelesscommunication module, or a global navigation satellite system (GNSS)communication module) or a wired communication module 194 (e.g., a localarea network (LAN) communication module or a power line communication(PLC) module). A corresponding one of these communication modules maycommunicate with the external electronic device via the first network198 (e.g., a short-range communication network, such as Bluetooth™,wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA))or the second network 199 (e.g., a long-range communication network,such as a cellular network, the Internet, or a computer network (e.g.,LAN or wide area network (WAN)). These various types of communicationmodules may be implemented as a single component (e.g., a single chip),or may be implemented as multi components (e.g., multi chips) separatefrom each other. The wireless communication module 192 may identify andauthenticate the electronic device 101 in a communication network, suchas the first network 198 or the second network 199, using subscriberinformation (e.g., international mobile subscriber identity (IMSI))stored in the subscriber identification module 196.

The antenna module 197 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 101. According to an embodiment, the antenna module197 may include one or more antennas, and, therefrom, at least oneantenna appropriate for a communication scheme used in the communicationnetwork, such as the first network 198 or the second network 199, may beselected, for example, by the communication module 190 (e.g., thewireless communication module 192). The signal or the power may then betransmitted or received between the communication module 190 and theexternal electronic device via the selected at least one antenna.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted orreceived between the electronic device 101 and the external electronicdevice 104 via the server 108 coupled with the second network 199. Eachof the electronic devices 102 and 104 may be a device of a same type as,or a different type, from the electronic device 101. According to anembodiment, all or some of operations to be executed at the electronicdevice 101 may be executed at one or more of the external electronicdevices 102, 104, or 108. For example, if the electronic device 101should perform a function or a service automatically, or in response toa request from a user or another device, the electronic device 101,instead of, or in addition to, executing the function or the service,may request the one or more external electronic devices to perform atleast part of the function or the service. The one or more externalelectronic devices receiving the request may perform the at least partof the function or the service requested, or an additional function oran additional service related to the request, and transfer an outcome ofthe performing to the electronic device 101. The electronic device 101may provide the outcome, with or without further processing of theoutcome, as at least part of a reply to the request. To that end, acloud computing, distributed computing, or client-server computingtechnology may be used, for example.

The electronic device according to various embodiments may be one ofvarious types of electronic devices. The electronic devices may include,for example, a portable communication device (e.g., a smart phone), acomputer device, a portable multimedia device, a portable medicaldevice, a camera, a wearable device, or a home appliance. According toan embodiment of the disclosure, the electronic devices are not limitedto those described above.

It should be appreciated that various embodiments of the presentdisclosure and the terms used therein are not intended to limit thetechnological features set forth herein to particular embodiments andinclude various changes, equivalents, or replacements for acorresponding embodiment. With regard to the description of thedrawings, similar reference numerals may be used to refer to similar orrelated elements. It is to be understood that a singular form of a nouncorresponding to an item may include one or more of the things, unlessthe relevant context clearly indicates otherwise. As used herein, eachof such phrases as “A or B,” “at least one of A and B,” “at least one ofA or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least oneof A, B, or C,” may include all possible combinations of the itemsenumerated together in a corresponding one of the phrases. As usedherein, such terms as “1st” and “2nd,” or “first” and “second” may beused to simply distinguish a corresponding component from another, anddoes not limit the components in other aspect (e.g., importance ororder). It is to be understood that if an element (e.g., a firstelement) is referred to, with or without the term “operatively” or“communicatively”, as “coupled with,” “coupled to,” “connected with,” or“connected to” another element (e.g., a second element), it means thatthe element may be coupled with the other element directly (e.g.,wiredly), wirelessly, or via a third element.

As used herein, the term “module” may include a unit implemented inhardware, software, or firmware, and may interchangeably be used withother terms, for example, “logic,” “logic block,” “part,” or“circuitry”. A module may be a single integral component, or a minimumunit or part thereof, adapted to perform one or more functions. Forexample, according to an embodiment, the module may be implemented in aform of an application-specific integrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software(e.g., the program 140) including one or more instructions that arestored in a storage medium (e.g., internal memory 136 or external memory138) that is readable by a machine (e.g., the electronic device 101).For example, a processor (e.g., the processor 120) of the machine (e.g.,the electronic device 101) may invoke at least one of the one or moreinstructions stored in the storage medium, and execute it, with orwithout using one or more other components under the control of theprocessor. This allows the machine to be operated to perform at leastone function according to the at least one instruction invoked. The oneor more instructions may include a code generated by a complier or acode executable by an interpreter. The machine-readable storage mediummay be provided in the form of a non-transitory storage medium. Wherein,the term “non-transitory” simply means that the storage medium is atangible device, and does not include a signal (e.g., an electromagneticwave), but this term does not differentiate between where data issemi-permanently stored in the storage medium and where the data istemporarily stored in the storage medium.

According to an embodiment, a method according to various embodiments ofthe disclosure may be included and provided in a computer programproduct. The computer program product may be traded as a product betweena seller and a buyer. The computer program product may be distributed inthe form of a machine-readable storage medium (e.g., compact disc readonly memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded)online via an application store (e.g., Play Store™), or between two userdevices (e.g., smart phones) directly. If distributed online, at leastpart of the computer program product may be temporarily generated or atleast temporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store,or a relay server.

According to various embodiments, each component (e.g., a module or aprogram) of the above-described components may include a single entityor multiple entities. According to various embodiments, one or more ofthe above-described components may be omitted, or one or more othercomponents may be added. Alternatively or additionally, a plurality ofcomponents (e.g., modules or programs) may be integrated into a singlecomponent. In such a case, according to various embodiments, theintegrated component may still perform one or more functions of each ofthe plurality of components in the same or similar manner as they areperformed by a corresponding one of the plurality of components beforethe integration. According to various embodiments, operations performedby the module, the program, or another component may be carried outsequentially, in parallel, repeatedly, or heuristically, or one or moreof the operations may be executed in a different order or omitted, orone or more other operations may be added.

FIG. 2 is a block diagram 200 illustrating the camera module 180according to various embodiments. Referring to FIG. 2, the camera module180 may include a lens assembly 210, a flash 220, an image sensor 230,an image stabilizer 240, memory 250 (e.g., buffer memory), or an imagesignal processor 260.

The lens assembly 210 focuses the light emitted from a scene that isphotographed. The flash 220 can be used to add light to the scene andenhance the light emitted from the scene. The image sensor 230 is thesubstrate that the images is captured on. The image stabilizer 240corrects for jitter that often occurs when the camera 180 is held byhand. The memory 250 at least temporarily stores the captured image. Theimage signal processor 260 performs various functions that enhance thequality of the captured image.

According to an embodiment, the camera module 180 may include aplurality of lens assemblies 210. In such a case, the camera module 180may form, for example, a dual camera, a 360-degree camera, or aspherical camera. The lens assembly 210 may collect light emitted orreflected from an object whose image is to be taken. The lens assembly210 may include one or more lenses. Some of the plurality of lensassemblies 210 may have the same lens attribute (e.g., view angle, focallength, auto-focusing, f number, or optical zoom), or at least one lensassembly may have one or more lens attributes different from those ofanother lens assembly. The lens assembly 210 may include, for example, awide-angle lens or a telephoto lens.

The flash 220 may emit light that is used to reinforce light reflectedfrom an object. According to an embodiment, the flash 220 may includeone or more light emitting diodes (LEDs) (e.g., a red-green-blue (RGB)LED, a white LED, an infrared (IR) LED, or an ultraviolet (UV) LED) or axenon lamp.

The image sensor 230 may obtain an image corresponding to an object byconverting light emitted or reflected from the object and transmittedvia the lens assembly 210 into an electrical signal. According to anembodiment, the image sensor 230 may include one selected from imagesensors having different attributes, such as a RGB sensor, ablack-and-white (BW) sensor, an IR sensor, or a UV sensor, a pluralityof image sensors having the same attribute, or a plurality of imagesensors having different attributes. Each image sensor included in theimage sensor 230 may be implemented using, for example, a chargedcoupled device (CCD) sensor or a complementary metal oxide semiconductor(CMOS) sensor.

The image stabilizer 240 may move the image sensor 230 or at least onelens included in the lens assembly 210 in a particular direction, orcontrol an operational attribute (e.g., adjust the read-out timing) ofthe image sensor 230 in response to the movement of the camera module180 or the electronic device 101 including the camera module 180. Thisallows compensating for at least part of a negative effect (e.g., imageblurring) by the movement on an image being captured. According to anembodiment, the image stabilizer 240 may sense such a movement by thecamera module 180 or the electronic device 101 using a gyro sensor (notshown) or an acceleration sensor (not shown) disposed inside or outsidethe camera module 180. According to an embodiment, the image stabilizer240 may be implemented, for example, as an optical image stabilizer.This is particularly useful for compensating for jitter that oftenresults from the camera module 180 being held by hand.

The memory 250 may store, at least temporarily, at least part of animage obtained via the image sensor 230 for a subsequent imageprocessing task. For example, if image capturing is delayed due toshutter lag or multiple images are quickly captured, a raw imageobtained (e.g., a Bayer-patterned image, a high-resolution image) may bestored in the memory 250, and its corresponding copy image (e.g., alow-resolution image) may be previewed via the display device 160.Thereafter, if a specified condition is met (e.g., by a user's input orsystem command), at least part of the raw image stored in the memory 250may be obtained and processed, for example, by the image signalprocessor 260. According to an embodiment, the memory 250 may beconfigured as at least part of the memory 130 or as a separate memorythat is operated independently from the memory 130.

The image signal processor 260 may perform one or more image processingwith respect to an image obtained via the image sensor 230 or an imagestored in the memory 250. The one or more image processing may include,for example, depth map generation, three-dimensional (3D) modeling,panorama generation, feature point extraction, image synthesizing, orimage compensation (e.g., noise reduction, resolution adjustment,brightness adjustment, blurring, sharpening, or softening). Additionallyor alternatively, the image signal processor 260 may perform control(e.g., exposure time control or read-out timing control) with respect toat least one (e.g., the image sensor 230) of the components included inthe camera module 180. An image processed by the image signal processor260 may be stored back in the memory 250 for further processing, or maybe provided to an external component (e.g., the memory 130, the displaydevice 160, the electronic device 102, the electronic device 104, or theserver 108) outside the camera module 180. According to an embodiment,the image signal processor 260 may be configured as at least part of theprocessor 120, or as a separate processor that is operated independentlyfrom the processor 120. If the image signal processor 260 is configuredas a separate processor from the processor 120, at least one imageprocessed by the image signal processor 260 may be displayed, by theprocessor 120, via the display device 160 as it is or after beingfurther processed.

According to an embodiment, the electronic device 101 may include aplurality of camera modules 180 having different attributes orfunctions. In such a case, at least one of the plurality of cameramodules 180 may form, for example, a wide-angle camera and at leastanother of the plurality of camera modules 180 may form a telephotocamera. Similarly, at least one of the plurality of camera modules 180may form, for example, a front camera and at least another of theplurality of camera modules 180 may form a rear camera.

FIG. 3 is a conceptual diagram illustrating operations of an electronicdevice 101 and an external electronic device 300 according to variousembodiments.

In various embodiments of the present disclosure, the electronic device101 may include an image sensor 321, an ISP 323, a memory 325, and arecognition module 327.

In various embodiments, the image sensor 321 may receive an image of anexternal scene. The image sensor 321 may provide a raw image 322 to theISP 323, and in some embodiments, the recognition module 327. The imagesensor 321 also provides a raw image 322 to the external electronicdevice 300. According to certain embodiments, due to bandwidthconsiderations, the image sensor 321 may generate a small raw image 326to provide to the external electronic device 300.

The external electronic device 300 may include a recognition module 331,an ISP 333, and a storage 335. The recognition module 331 may be alogical module and may be implemented as a processor of the externalelectronic device 300. Also, the ISP 333 may be implemented as aprocessor of the external electronic device 300. For example, theprocessor of the external electronic device 300 may perform bothrecognition and image processing. Although not shown, the electronicdevice 101 may include a communication module (e.g., the communicationmodule 190 of FIG. 1) capable of transmitting and receiving data to andfrom the external electronic device 300.

Also, the external electronic device 300 may include a communicationmodule capable of transmitting and receiving data to and from theelectronic device 101. According to a certain embodiment, the electronicdevice 101 may also include a recognition module 327. The recognitionmodule 327 equipped in the electronic device 101 may be configured toperform the same function as at least some of functions of therecognition module 331 of the external electronic device 300. Forexample, the recognition module 327 may be hardware configured torecognize a face in an image and may be used for the purpose ofrecognizing a face more simply and quickly than the external electronicdevice 300 (e.g., a server).

In various embodiments, the image sensor 321 (e.g., the image sensor 230of FIG. 2) may acquire an image for an external scene and generate acorresponding raw image 322. The raw image 322 may be generated invarious formats such as a Bayer format, a format processed by a colorfilter array (CFA) pattern, a format of a layer structure created bysensing all three colors in one pixel, or a format created by acquiringdifferent parallax values in one pixel. The image sensor 321 may deliverthe raw image 322 to the ISP 323 (e.g., the ISP 260 of FIG. 2) and/orthe recognition module 327.

In various embodiments, the image sensor 321 may generate a small rawimage 326 by reducing the volume, or bit size, of the raw image 322. Forexample, the image sensor 321 may generate the small raw image 326 fromthe raw image 322 by using various down-scale or down-samplingtechniques. For example, by performing at least one of adjusting theresolution of the raw image 322, selecting at least some of a pluralityof frequency bands, such as the lower frequency bands, or selecting atleast one of a plurality of bit plane levels, the image sensor 321 maygenerate the small raw image 326 having a size smaller than a data sizeof the raw image 322. For example, by extracting a low frequency bandfrom the raw image 322, the image sensor 321 may generate the small rawimage 326. For example, by selecting only some of a plurality of bitplane levels of the raw image 322, the image sensor 321 may generate thesmall raw image 326.

The image sensor 321 may transmit the small raw image 326 to theexternal electronic device 300 through the communication module, savingbandwidth. The small raw image 326 may be an image including at leastpart of information of the raw image 322 and being smaller in volumethan the raw image 322. In case of transmitting the small raw image 326,instead of the raw image 322, to the external electronic device 300, itis possible to provide an image to the external electronic device 300faster through transmission of smaller volume. In another embodiment, aprocessor (e.g., the processor 120) of the electronic device 101,instead of the image sensor 321, may generate the small raw image 326and transmit the generated small raw image 326 to the externalelectronic device 300 through the communication module.

In various embodiments, the image sensor 321 may transmit the raw image322 in a compressed state to the ISP 323, the external electronic device300, and/or the recognition module 327. The image sensor 321 maycompress the raw image 322 for partial processing and may store thecompressed raw image 322 in an internal memory thereof.

In various embodiments, the recognition module 331 of the externalelectronic device 300 may acquire the small raw image 326 through thecommunication module and may perform segmentation for dividing the smallraw image 326 into at least one image segment. The recognition module331 is capable of identifying objects in the image and segmenting theimage as a result of the object identification. For example, therecognition module 331 may perform segmentation processing on the smallraw image 326 and, based on a segmentation processing result, mayidentify at least one image segment from the small raw image 326. Therecognition module 331 may recognize at least one image segment byapplying an object recognition algorithm or a texture recognitionalgorithm to the image segment. The recognition module 331 may recognizeat least one image segment by using various recognition algorithms orusing a recognition algorithm acquired through machine learning or deeplearning.

For example, the recognition module 331 of the external electronicdevice 300 may acquire information associated with an image segment,such as information indicating that the pixel coordinate values (100,101), (100, 102), (102, 102) and (102, 103) correspond to person'steeth. Such pixel coordinate values may correspond to pixel coordinatevalues of the raw image 322. In addition, the recognition module 331 mayacquire classification information such as, for example, informationindicating that the small raw image 326 belongs to a category of “aperson located on the street”.

The recognition module 331 may acquire the classification information byusing the recognition result or using a color distribution in the smallraw image 326 without a recognition process. The recognition module 331may generate correction area information 332 that includes at least oneof the acquired information associated with at least one image segmentor the acquired classification information. Also, the recognition module331 may transmit the correction area information 332 to the electronicdevice 101. Then the ISP 323 of the electronic device 101 may correctthe raw image 322 by using the correction area information 332, and thusa corrected image 324 may be generated. The corrected image 324 mayhave, for example, a YUV format. Also, the corrected image 324 may bestored in the memory 325. Also, the corrected image 324 may becompressed, for example, in accordance with the JPEG method, and thecompressed image may be stored in the memory 325. According to a certainembodiment, the correction area information 322 may be generated by therecognition module 327 of the electronic device 101 and then deliveredto the ISP 323.

In various embodiments, the raw image 322 provided from the image sensor321 may be transmitted to the external electronic device 300 separatelyfrom the small raw image 326. Using the raw image 322, the externalelectronic device 300 may generate other correction area information.That is, using the raw image 322 having a size greater than the smallraw image 326, the external electronic device 300 (e.g., the ISP 333thereof) may generate correction area information different from thecorrection area information in case of using the small raw image 326.This may be referred to as extended correction area information. Sincethe raw image 322 may contain much more information than the small rawimage 326, the external electronic device 300 may generate more detailedcorrection area information. In various embodiments, the externalelectronic device 300 (e.g., the ISP 333 thereof) may generate theextended correction area information directly from the raw image 322.Also, using the correction area information previously generated fromthe small raw image 326 and also using the raw image 322, the externalelectronic device 300 (e.g., the ISP 333 thereof) may generate theextended correction area information.

In various embodiments, since the raw image 322 is greater in volumethan the small raw image 326, the small raw image 326 may be firsttransmitted to the external electronic device 300 and then the raw image322 may be transmitted to the external electronic device 300. Forexample, the raw image 322 may be transmitted to the external electronicdevice 300 while the ISP 323 performs the correction on the raw image322. The raw image 322 may be uploaded to the external electronic device300 in a state of being generated by the image sensor 321 or may beuploaded as a preprocessed image in which lens distortion compensationor noise removal has been performed. Such preprocessing may be performedin the external electronic device 300. The external electronic device300 may perform demosaic processing (a digital image process used toreconstruct a full color image from the incomplete color samples outputfrom an image sensor overlaid with a color filter array (CFA). It isalso known as CFA interpolation or color reconstruction), image formatmodification, or preprocessing to increase an object recognition rate.The ISP 333 of the external electronic device 300 may correct thereceived raw image 322 by using the previously generated correction areainformation 332 or using the extended correction area information. Theraw image 322 may have a higher resolution than that of the small rawimage 326, so that the ISP 333 of the external electronic device 300 mayacquire more detailed extended correction area information from a highquality image. The ISP 333 may generate the extended correction areainformation by using both the previously generated correction areainformation and the raw image 322. The ISP 333 may acquire a highquality image 334 by correcting the raw image 322 through the extendedcorrection area information. The high quality image 334 may be stored inthe storage 335 of the external electronic device 300 and downloaded tothe electronic device 101.

The electronic device according to various embodiments may be one ofvarious types of electronic devices. The electronic devices may include,for example, a portable communication device (e.g., a smart phone), acomputer device, a portable multimedia device, a portable medicaldevice, a camera, a wearable device, or a home appliance. According tovarious embodiments, the electronic device is not limited to thosedescribed above.

It should be appreciated that various embodiments of the presentdisclosure and the terms used therein are not intended to limit thetechnological features set forth herein to particular embodiments andinclude various changes, equivalents, or replacements for acorresponding embodiment. With regard to the description of thedrawings, similar reference numerals may be used to refer to similar orrelated elements. It is to be understood that a singular form of a nouncorresponding to an item may include one or more of the things, unlessthe relevant context clearly indicates otherwise. As used herein, eachof such phrases as “A or B”, “at least one of A and B”, “at least one ofA or B”, “A, B, or C”, “at least one of A, B, and C”, and “at least oneof A, B, or C” may include all possible combinations of the itemsenumerated together in a corresponding one of the phrases. As usedherein, such terms as “1st” and “2nd”, or “first” and “second” may beused to simply distinguish a corresponding component from another, anddoes not limit the components in other aspect (e.g., importance ororder). It is to be understood that if an element (e.g., a firstelement) is referred to, with or without the term “operatively” or“communicatively”, as “coupled with”, “coupled to”, “connected with”, or“connected to” another element (e.g., a second element), it means thatthe element may be coupled with the other element directly (e.g.,wiredly), wirelessly, or via a third element.

As used herein, the term “module” may include a unit implemented inhardware, or hardware programmed with software, and may interchangeablybe used with other terms, for example, “logic”, “logic block”, “part”,“circuitry”. A module may be a single integral component, or a minimumunit or part thereof, adapted to perform one or more functions. Forexample, according to an embodiment, the module may be implemented in aform of an application-specific integrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software(e.g., the program 140) including one or more instructions that arestored in a storage medium (e.g., internal memory 136 or external memory138) that is readable by a machine (e.g., the electronic device 101).For example, a processor (e.g., the processor 120) of the machine (e.g.,the electronic device 101) may invoke at least one of the one or moreinstructions stored in the storage medium, and execute it, with orwithout using one or more other components under the control of theprocessor. This allows the machine to be operated to perform at leastone function according to the at least one instruction invoked. The oneor more instructions may include a code generated by a complier or acode executable by an interpreter. The machine-readable storage mediummay be provided in the form of a non-transitory storage medium. Wherein,the term “non-transitory” simply means that the storage medium is atangible device, and does not include a signal (e.g., an electromagneticwave), but this term does not differentiate between where data issemi-permanently stored in the storage medium and where the data istemporarily stored in the storage medium.

According to an embodiment, a method according to various embodiments ofthe disclosure may be included and provided in a computer programproduct. The computer program product may be traded as a product betweena seller and a buyer. The computer program product may be distributed inthe form of a machine-readable storage medium (e.g., compact disc readonly memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded)online via an application store (e.g., Play Store™), or between two userdevices (e.g., smart phones) directly. If distributed online, at leastpart of the computer program product may be temporarily generated or atleast temporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store,or a relay server.

According to various embodiments, each component (e.g., a module or aprogram) of the above-described components may include a single entityor multiple entities. According to various embodiments, one or more ofthe above-described components may be omitted, or one or more othercomponents may be added. Alternatively or additionally, a plurality ofcomponents (e.g., modules or programs) may be integrated into a singlecomponent. In such a case, according to various embodiments, theintegrated component may still perform one or more functions of each ofthe plurality of components in the same or similar manner as they areperformed by a corresponding one of the plurality of components beforethe integration. According to various embodiments, operations performedby the module, the program, or another component may be carried outsequentially, in parallel, repeatedly, or heuristically, or one or moreof the operations may be executed in a different order or omitted, orone or more other operations may be added.

FIG. 4 is a block diagram illustrating an electronic device and anexternal electronic device according to various embodiments.

The electronic device 400 includes an image sensor 411 that takes aphotograph of a scene, thereby resulting in an image. The secondprocessor 412 includes a raw image processor for generating a version ofthe raw image with reduced bandwidth, either a compressed raw image or asmall raw image. In some embodiments, the small raw image is transmittedto the external electronic device 470. External electronic device 470includes an image recognition module that identifies objects in theimage and determines a reliability measure for each of the identifiedobjects. The user input recognition module 472, prompts the user toinput the objects that are seen in the image. The image recognitionmodule 473 locates each of the objects that are input by the user. Theobjects that are among both groups have reliability measured increased.

The electronic device 400 (e.g., the electronic device 101) according tovarious embodiments may include a camera module 410, a display 420, afirst processor 430, a first memory 440, an audio device 450, and anaudio output device 460. The camera module 410 according to variousembodiments may include an image sensor 411, a second processor 412, anda second memory 418. The second processor 412 according to variousembodiments may include a raw image processing module 413, an ISP 416,and an encoder 417. At least some of operations performed by the secondprocessor 412 may be performed by the first processor 430. The raw imageprocessing module 413, the ISP 416, and the encoder 417 may be logicalmodules, and thus operations thereof may be performed by the secondprocessor 412 (e.g., the processor 120). In another embodiment, at leastone of the raw image processing module 413, the ISP 416, and the encoder417 may be implemented as hardware within the second processor 412.

Although not shown, the electronic device 400 may include acommunication module (e.g., the communication module 190 of FIG. 1) fordata communication with an external electronic device 470. Also, theexternal electronic device 470 may include a communication module fordata communication with the electronic device 400.

In various embodiments, the image sensor 411 (e.g., the image sensor 230of FIG. 2 or the image sensor 321 of FIG. 3) may acquire various rawimages of things for photography. The image sensor 411 may acquirevarious types of raw images depending on a color filter array (CFA)pattern. The image sensor 411 may acquire a raw image of a dual pixel(DP) structure (or 2PD) including different parallax values (or a phasedifference) in one pixel. The image sensor 411 may include a pluralityof image sensors having the same or different characteristics (e.g., adual sensor such as RGB & RGB, RGB & Mono or Wide & Tele, or an arraysensor having two or more sensors attached). Using these image sensors,it is possible to acquire one or more raw images with respect to onescene. The acquired raw image may be stored in the second memory 418(e.g., DRAM) as it is or after additional processing.

The raw image acquired according to various embodiments may be formed invarious formats (e.g., Bayer format). The raw image may be representedby one color of red (R), green (G) and blue (B) per pixel and in abit-depth from 8 to 16 bits. A variety of CFA patterns may be applied tothe raw image. The raw image may have a layer structure includinginformation of a plurality of colors (e.g., two or more colors of R, Gand B) for one pixel. Depending on various configurations of the imagesensor, the raw image may include parallax (phase difference)information as well as color information. Information related to imageshooting (e.g., time, location, illumination, etc.) may be generated asmetadata and stored in association with the raw image. For example, thesecond processor 412 may acquire metadata for the raw image through theimage sensor 411. The metadata acquirable through the image sensor 411may include a focal length, an auto focus area, an orientation in caseof shooting, a color space, an exposure time, and the like. In addition,the metadata may include location information on image shooting, etc.,which can be obtained through another sensor (e.g., a GPS sensor)different from the image sensor.

In various embodiments, the raw image processing module 413 may performvarious types of processing on the raw image acquired from the imagesensor 411. For example, the raw image processing module 413 may performlens distortion compensation or noise removal on the raw image.

The raw image processing module 413 according to various embodiments mayinclude a small raw image generation module 414 and a compression module415. The small raw image generation module 414 may generate the smallraw image from the raw image by using various down-scale techniques(e.g., operations of reducing a size or lowering a resolution) ordown-sampling techniques (e.g., operations of taking only one or some ofsamples). The compression module 415 may compress the raw image or thesmall raw image by using various compression algorithms, and may storethe compressed raw image or the compressed small raw image in the secondmemory 418. The small raw image may be temporarily or non-temporarilystored in the second memory 418. A communication module (not shown,e.g., the communication module 190) of the electronic device 400 maytransmit the small raw image stored in the second memory 418 to theexternal electronic device 470.

In various embodiments, the ISP 416 (e.g., the ISP 260 of FIG. 2 or theISP 423 of FIG. 3) may perform image processing on the raw image storedin the second memory 418. For example, using recipe information (e.g.,image segment, layer, vector, scene category, etc.) acquired from theexternal electronic device 470 via the communication module, the ISP 416may perform various kinds of processing (e.g., correction) on the rawimage. According to one embodiment, the ISP 416 may transmit the rawimage or the small raw image to the external electronic device 470through the communication module.

External electronic device 470 includes an image recognition module 473that identifies objects in the image and determines a reliabilitymeasure for each of the identified objects. The user input recognitionmodule 472, prompts the user to input the objects that are seen in theimage. The image recognition module 473 locates each of the objects thatare input by the user. The objects that are among both groups havereliability measured increased.

The external electronic device 470 provides type of objects andlocations that are identified with high reliability. Then the ISP 416may receive the type of objects and locations from the externalelectronic device 470 through the communication module and process theraw image. The ISP 416 may compress the processed raw image in the formof JPEG for example, and store the compressed raw image in the firstmemory 440.

In various embodiments, the encoder 417 may encode the raw imageprocessed by the ISP 416 to generate an image file (e.g., JPEG MPEG360-degree panorama, etc.), and may store the generated image file inthe first memory 440.

In various embodiments, the first processor 430 (e.g., the processor 120of FIG. 1) may be electrically connected to the camera module 410, thedisplay 420, the first memory 440, the input device 450, the audiooutput device 460, and the communication module (not shown), control atleast one of them, and perform various data processing and operations.

According to various embodiments, the first processor 430 may transmitan image (e.g., corresponding to an image being displayed through thedisplay 420) stored in the electronic device 400 to the externalelectronic device 470 through the communication module such that theexternal electronic device 470 recognizes an object (e.g., person, sky,balloon, grass) in the image. For example, the first processor 430, orthe second processor 412 under the control of the first processor 430,may acquire the raw image, the small raw image, the compressed rawimage, or the compressed small raw image from the second memory 418 andtransmit the acquired image to the external electronic device 470through the communication module. In another example, the firstprocessor 430 may acquire an image file, corresponding to the imagebeing displayed via the display 420, from the first memory 440 andtransmit the image file to the external electronic device 470.

According to various embodiments, the first processor 430 may transmit auser input, acquired in the electronic device 400, to the externalelectronic device 470 through the communication module such that theexternal electronic device 470 recognizes a word corresponding to theobject from the user input. In various embodiments, the user input maybe acquired while the image corresponding to the image transmitted tothe external electronic device 470 is being displayed. For example, theuser input may be a user's utterance (e.g., “Make the sea color muchbluer.”) acquired through the microphone while a certain photo is beingprovided through the display 420. In various embodiments, the firstprocessor 430 may perform preprocessing (e.g., noise removal orsuppression, conversion to text, automatic gain control (AGC), adaptiveecho canceller, etc.) on the user's utterance in order to increase avoice recognition rate in the external electronic device 470 or reducethe size of data to be transmitted to the external electronic device470. Then the preprocessed utterance may be transmitted from theelectronic device 400 to the external electronic device 470. In variousembodiments, the user input may be text obtained from the keyboard ofthe electronic device 400 or from the touch-sensitive display 420, andmay be transmitted to the external electronic device 470 via thecommunication module.

The first processor 430 according to various embodiments may receiveobject matching information from the external electronic device 470 inresponse to the transmission of the user input and the image (or imagefile). In various embodiments, the object matching information may beinformation associated with a specific object matched with a specificword (e.g., “sea” as a grammatical object corresponding to an object inthe user utterance, “Make the sea color much bluer.”) recognized fromthe user input among objects in the image. That is, the object matchinginformation may be information for identifying an object, which isindicated as an editing target by the user, selected from among objectsin the displayed image. For example, the object matching information mayinclude pixel coordinate values indicating the location of the selectedobject in the image. Additionally, the first processor 430 may receiveinformation about an operation (e.g., an editing method) correspondingto a user's intent (e.g., command) from the external electronic device470 in response to the transmission of the user input and the image (orimage file) according to various embodiments.

The first processor 430 or the second processor 412 (e.g., the ISP 416)according to various embodiments may edit an image, based on informationreceived from the external electronic device 470 in response to thetransmission of the user input and the image (or image file). Forexample, the first processor 430 or the second processor 412 may acquirean image (e.g., corresponding to an image being displayed through thedisplay 420) stored in the electronic device 400 through the memory(e.g., the first memory 440 or the second memory 418), and may identifya specific segment in the acquired image, based on the object matchinginformation received from the external electronic device 470. Also, thefirst processor 430 may identify a processing method for the specificsegment, based on operation information received from the externalelectronic device 470. Then, the first processor 430 may edit the imageby processing the specific segment, based on the identified processingmethod. The first processor 430 may display the edited image via thedisplay 420 and store the edited image in the memory (e.g., the firstmemory 440 or the second memory 418).

According to various embodiments, such image editing may be performed byan external electronic device in response to the reception of the userinput and the image (or image file) from the electronic device 400. Forexample, the external electronic device 470 (e.g., the ISP 476), oranother external electronic device (not shown, e.g., an image editingserver) functionally connected thereto, may perform the image editingand transmit an edited result to the electronic device 400 directly orvia the external electronic device 470.

The first processor 430 according to various embodiments may request auser feedback for the image editing. For example, the first processor430 may output a pop-up message having content of inquiring about theaccuracy of editing (e.g., “Did the sea color become much bluer?”)through the display 420, or may output a corresponding voice through theaudio output device 460 (e.g., the speaker). In addition, the firstprocessor 430 may display the edited image segment to be distinguishedfrom the other image segments. For example, various techniques ofvisually distinguishing the image segments from each other, such asemphasizing the outline of the edited image segment, may be applied tovarious embodiments. Such visual distinction may allow the user toidentify the edited image segment and judge whether the image editinghas been done in accordance with his or her intention. The firstprocessor 430 may obtain a user feedback on the inquiry through thedisplay 420 or the input device 450 (e.g., the microphone), and maytransmit the user feedback to the external electronic device 470 to useit in improving the performance of image recognition. According tovarious embodiments, the external electronic device 470 may adjust therecognition reliability by using the user feedback received from theelectronic device 400.

If any object other than the objected intended by the user has beenedited as a result of checking the user feedback, the first processor430 according to various embodiments may request the user to directlydesignate an editing target. For example, the first processor 430 mayoutput a pop-up message (e.g., “Touch the sea.”) requesting the user todesignate an editing target through the display 420 or may output acorresponding voice through the audio output device 460 (e.g., thespeaker). The first processor 430 may obtain a user feedback on thedirect designation request through the display 420 or the input device(e.g., the microphone), and may transmit the user feedback to theexternal electronic device 470.

The first processor 430 according to various embodiments may receive,from the external electronic device 470, a message of failing to findthe editing target in response to the transmission of the user input andthe image (or image file). Then, the first processor 430 may request theuser to directly designate the editing target. For example, the firstprocessor 430 may output a pop-up message (e.g., “Touch the sea.”)requesting the user to designate an editing target through the display420 or may output a corresponding voice through the audio output device460 (e.g., the speaker). The first processor 430 may obtain a userfeedback on the direct designation request through the display 420 orthe input device 450 (e.g., the microphone), and may transmit the userfeedback to the external electronic device 470.

The external electronic device 470 according to various embodiments maybe implemented as a cloud server. The external electronic device 470 mayperform functions of a network management for servers constituting acloud system and for electronic devices (e.g., the electronic device400) connectable to the cloud system, a cloud service managementassociated with providable services and rights, and a storagemanagement. The external electronic device 470 may include a processor471, a database 478, a raw image storage 479, and a learning imagestorage 480. The processor 471 according to various embodiments mayinclude a preprocessing module 474, a user input recognition module 472,an image recognition module 473, an encoder 475, and an ISP 476. Atleast some of operations performed by the first processor 430 or thesecond processor 412 of the electronic device 400 may be performed bythe processor 471 of the external electronic device 470. Thepreprocessing module 474, the user input recognition module 472, theimage recognition module 473, the encoder 475, and the ISP 476 may belogical modules, and thus the operations thereof may be performed by theprocessor 471 or the first or second processor 430 or 412 of theelectronic device 400. In another embodiment, at least one of thepreprocessing module 474, the user input recognition module 472, theimage recognition module 473, the encoder 475, and the ISP 476 may beimplemented as hardware within the processor 471 of the externalelectronic device 470 or as hardware within the first or secondprocessor 430 or 412 of the electronic device 400.

The user input recognition module 472 according to various embodimentsmay receive a user input (e.g., utterance or text) from the electronicdevice 400 via a communication module (not shown) of the externalelectronic device 470. The user input recognition module 472 may acquirevarious kinds of meaningful information (e.g., a grammatical subject, agrammatical object, a command, etc.) from the user input. According toone embodiment, the user input recognition module 472 may convertutterance to text. For example, the user input recognition module 472may include an acoustic model and a language model. For example, theacoustic model may include information related to speech, and thelanguage model may include information about phoneme units and acombination thereof. The user input recognition module 472 may convert auser's utterance into text by using the utterance-related informationand the phonemic-related information. Information about the acousticmodel and the language model may be stored, for example, in thedatabase. According to one embodiment, the user input recognition module472 may acquire meaningful information from text (e.g., the user inputconverted from utterance to text by the electronic device 400 or theuser input recognition module 472) by performing natural languageunderstanding (NLU). For example, the user input recognition module 472may obtain a grammatical object by dividing text into sentencecomponents through a syntactic analysis. Also, the user inputrecognition module 472 may understand a user's intention (e.g., command)by performing a grammatical analysis or a semantic analysis.

The user input recognition module 472 according to various embodimentsmay generate an object list from information acquired as a result ofrecognizing the user input (e.g., a grammatical object “sea” from theuser utterance, “Make the sea color much bluer.”). In variousembodiments, objects contained in the object list may be editingtargets.

The user input recognition module 472 according to various embodimentsmay generate operation information from information acquired as a resultof user input recognition. For example, from the user utterance, “Makethe sea color much bluer.”, the operation information may include “make. . . bluer” corresponding to a user intention (command) and alsoinclude “sea” and “much” as parameters necessary for expressing the userintention. In various embodiments, using the operation information, theelectronic device (e.g., the electronic device 400 or the externalelectronic device 470) may perform image processing on an objectdesignated as the editing target.

The preprocessing module 474 according to various embodiments maypreprocess an image (e.g., a raw image, a small raw image) or an imagefile received from the electronic device 400 and then sends it to theimage recognition module 473 or the ISP 476. For example, preprocessingmay include an operation of acquiring the raw image by decompressing theimage file, an operating of performing demosaic processing, an operationof transforming an image format into YUV, or the like.

The image recognition module 473 according to various embodiments mayreceive the image (e.g., the raw image, the small raw image) from theelectronic device 400 through the communication module of the externalelectronic device 470 or further through the preprocessing module 474.

The image recognition module 473 according to various embodiments mayperform an operation of analyzing various kinds of meaningfulinformation (e.g., object recognition, velocity vector representing thevelocity of a specific object in an image, face recognition,segmentation, scene parsing, etc.) from the received image. The imagerecognition module 473 may perform an operation of generating, storing,or transmitting the analysis result in association with the image. Theanalysis result may include recipe information such as image segment,layer, vector, or scene category, and may be utilized in imageprocessing by the ISP 476.

The image recognition module 473 according to various embodiments mayacquire various kinds of meaningful information (e.g., locationinformation such as pixel coordinate values about each image segment,identification information of an object associated with each imagesegment, reliability associated with object recognition, etc.) from thereceived image by using a recognition algorithm obtained by applyingmachine learning or deep learning to the learning image storage 480.According to a certain embodiment, the image recognition module 473 mayperform object recognition in an image, based on user information. Forexample, the user information may be acquired from images (e.g., photosof family and relatives, photos of residence, etc.) registered in thedatabase 478.

The image recognition module 473 according to various embodiments mayidentify a specific object designated as an editing target by the user,by comparing information acquired as the image recognition result withinformation acquired as the speech recognition result. For example, theimage recognition module 473 may receive information about an object (agrammatical object) from the user input recognition module 472, and mayidentify, as an editing target, a specific object matched with thegrammatical object among objects identified through image recognition.The image recognition module 473 may transmit object matchinginformation about the identified object to the electronic device 400 orthe ISP 476. Additionally, the image recognition module 473 may identifythe object, based on the reliability associated with the recognition ofthe object acquired as a result of image recognition. For example, animage is divided into one or more image segments, each of which may berecognized as one or more objects. Also, the object may have reliabilityassigned thereto in connection with recognition. For example, a firstimage segment may be recognized with a probability (i.e., reliability)of ‘sea’ being 80% (first priority) and a probability of ‘sky’ being 20%(second priority). A second image segment may be recognized with aprobability of ‘sky’ being 80% (first priority) and a probability of‘sea’ being 20% (second priority). In this situation, when the editingtarget recognized from the user input is ‘sea’, the first image segmentmay be determined as the editing target because the editing target iscloser to the first image segment in terms of reliability(probabilistically) than to the second image segment.

The image recognition module 473 according to various embodiments maymanage objects having similar meanings (e.g., a bookstore, a bookshop,and a book stall) as one set of synonyms. Based on such sets ofsynonyms, the image recognition module 473 may identify a specificobject designated as the editing target by the user among objectsrecognized from the image. For example, if there is ‘a bookstore’ amongobjects recognized from the image, but if the user input has no‘bookstore’ and has only synonym ‘bookshop’ or ‘book stall’, the‘bookstore’ may be identified as a target designated by the user.

The image recognition module 473 according to various embodiments mayadjust the reliability associated with the recognition of an objectacquired as a result of image recognition, based on a reliability-basedmatching result. For example, one image segment may be recognized as aplurality of objects, and priorities may be assigned to respectiveobjects according to reliability in an object list corresponding to oneimage segment. In one embodiment, the reliability adjustment may beperformed according to the ranking assigned to “an object matched withthe editing target recognized through user input recognition” amongobjects. For example, when the top-rank object is matched with theediting target, the image recognition module 473 may increase (i.e.,upwardly adjust) the reliability of the object. If the next-rank (e.g.,second priority) object is matched with the editing target, the imagerecognition module 473 may decrease (i.e., downwardly adjust) thereliability of an object having higher rank (e.g., the first priorityobject) and increase the reliability of the corresponding object. Forexample, a certain image segment may be recognized as a ‘sun’ or a‘torch’, and the probability of being ‘sun’ may be higher. In thissituation, if the user input has a ‘sun’, the reliability of the object‘sun’ having the first priority may be increased. If the user input hasa ‘torch’, the reliability of the object ‘sun’ having the first prioritymay be decreased and the reliability of the object ‘torch’ having thesecond priority may be increased. If an object with lower rank ismatched with the editing target, the image recognition module 473 mayperform an operation of changing the ranking. For example, in case ofthe second priority matching, it is possible to change the first andsecond priorities of objects to each other.

The image recognition module 473 according to various embodiments mayadjust the reliability associated with the recognition of an objectacquired as a result of image recognition, based on a user feedback fora matching result. For example, if the user feedback for the firstpriority matching indicates a correct answer, the image recognitionmodule 473 may increase the reliability of the first priority object. Ifthe feedback for the second priority matching indicates a correctanswer, the image recognition module 473 may decrease the reliability ofthe first priority object and increase the reliability of the secondpriority object. If the feedback for the second priority matchingindicates a correct answer, the image recognition module 473 may changethe ranking. If the feedback for matching indicates an incorrect answer,the image recognition module 473 may decrease the reliability or rankingof the object.

According to various embodiments, if the user feedback for the matchingresult indicates an incorrect answer or is not matched up to apredetermined rank (e.g., second priority), the image recognition module473 may transmit, to the electronic device 400 through the communicationmodule of the external electronic device 470, a message of requestingthe user to directly designate the editing target. Based on a feedbackof this request, the image recognition module 473 may adjust thereliability.

As an example, if there is an object designated by the user in theobject list, the reliability of the object may be increased (e.g.,adjusted upward to exceed the first priority object, thus prioritiesbeing changed).

As another example, a certain image region may be recognized as a ‘sun’,‘torch’, or ‘streetlight’, in which ‘sun’ has the highest priority and‘streetlight’ has the lowest priority. In this situation, if the userinput has a ‘street light (third priority)’ other than a ‘sun (firstpriority)’ or ‘torch (second priority)’, the reliability may be adjustedupward (e.g., 100%) for the object ‘streetlight’ as the object closestto the image segment.

As still another example, if there is no object designated by the userin the object list (e.g., if the user input has neither sun, nor torch,nor streetlight, and has a word which does not exist in the objectlist), the word designated by the user may be added as an objectrecognizable in connection with the image segment, and the reliabilitythereof may be set to a predetermined value or less (e.g., 50% or less).

As yet another example, the absence of an object designated by the userin the object list may mean a recognition error of user input. Forexample, when the user pronounces ‘sea’, a similar pronunciation otherthan ‘sea’ may be recognized due to an error of speech recognition. Thismay result in the absence of an object designated by the user in theobject list. In this case, the image recognition module 473 may transmita re-request of the user input to the electronic device 400 through thecommunication module of the external electronic device 470. Then, theimage recognition module 473 may adjust reliability, based on a userreaction to the re-request.

As further another example, a certain image segment in the image may notbe recognized as an object. In this situation, when the user selectssuch an ‘unidentified image segment’, an object designated by the usermay be determined as an object of the image segment, and the reliabilitythereof may be set as a predetermined value.

When the user feedback for a matching result indicates an incorrectanswer, the image recognition module 473 according to variousembodiments may determine whether the incorrect answer is intentional.If so, the image recognition module 473 may transmit a warning messageto the electronic device 400 via the communication module of theexternal electronic device 470. For example, when an incorrect answerrepeatedly happens more than a predetermined number of times, a warningmessage that indicates a possibility of decreased recognition rate maybe sent to the user. According to various embodiments, operations forimage recognition (i.e., an image recognition model) in the imagerecognition module 473 may be personalized to be adapted to the user.Thus, if the feedback is intentionally incorrect, the image recognitionrate for the user may be continuously reduced. If the recognition rateof some objects is lower than the recognition rate of a commonrecognition engine, a user's personalized recognition engine may bereplaced with the common recognition engine of the initialization value.

The image recognition module 473 according to various embodiments maystore the adjusted reliability in the learning image storage 480 so thatimage recognition is performed using the adjusted reliability.

The ISP 476 according to various embodiments may perform imageprocessing based on recipe information acquired through imagerecognition. The ISP 476 may receive additional information (e.g., afeature vector representing the feature of an object or a part thereof(e.g., hair)) corresponding to the recipe information from the database478 and use the received additional information in image processing. Theprocessed image may be transmitted to the electronic device 400 or theencoder 475 or may be stored in the raw image storage 479. The imageprocessing may include functions such as a white balance, a coloradjustment, a noise reduction, a sharpening, and a detail enhancement.These functions may be performed for each image segment, based on therecipe information.

The encoder 475 according to various embodiments may generate an imagefile (e.g., JPEG MPEG 360-degree panorama, etc.) by encoding the rawimage processed by the ISP 476. The image file generated by the encoder475 may be transmitted to the electronic device 400 via thecommunication module of the external electronic device 470 or may bestored in the raw image storage 479.

FIG. 5 is a flow diagram illustrating operations of a processoraccording to various embodiments.

The operations shown in FIG. 5 may be executed by the processor 120 ofFIG. 1. In addition, such operations may be executed through at leastone of the processors 471, 412, and 430 of FIG. 4.

According to various embodiments, at operation 510, the processor mayreceive an image. For example, this image identified by the processormay be a raw image acquired through the image sensor (e.g., 411 in FIG.4) or a small raw image acquired through the raw image processing module(e.g., 413 in FIG. 4). In addition, the image may be a decompressedversion of an image file acquired through the second memory 418 or thefirst memory 430. In various embodiments, objects in the image may berecognized using various image recognition schemes (e.g., the imagerecognition module 473). For example, the image may be divided into aplurality of image segments, and each image segment may be recognized asone or more objects by various image recognition schemes.

According to various embodiments, at operation 520, the processor mayacquire one or more reliabilities associated with the recognition of oneor more first objects recognized from the image. For example, the one ormore first objects may be recognized as objects corresponding to one ofthe image segments, and the one or more reliabilities associated withthe recognition of each of the one or more first objects may be acquiredusing various image recognition schemes (e.g., image recognition module473).

According to various embodiments, at operation 530, the processor mayacquire a user input from at least one of, for example, the microphone,the keyboard, or the touch-sensitive display. Words in the user inputmay be recognized through various character recognition schemes (e.g.,the user input recognition module 472). In certain embodiments, the usermay be prompted to input which objects appear in the image.

According to various embodiments, at operation 540, the processor mayacquire one or more second objects (e.g., a word corresponding to agrammatical object among sentence components) from words included in theuser input.

According to various embodiments, at operation 550, when there is atleast one object corresponding to the one or more second objects amongthe one or more first objects, the processor may adjust at least onereliability corresponding to the at least one object. For example,suppose that the one or more first objects include a first priorityobject and a second priority object the priorities of which aredetermined in order of reliability. When the first priority objectcorresponds to the second object, the reliability of the first priorityobject may be increased (i.e., adjusted upward). When the secondpriority object corresponds to the second object, the reliability of thefirst priority object may be decreased (i.e., adjusted downward) and thereliability of the second object may be increased (i.e., adjustedupward).

According to various embodiments, at operation 560, the processor mayrecognize the one or more first objects by using the image recognition,at least based on the adjusted at least one reliability. For example,the second priority object before the reliability adjustment may berecognized as the first priority object after the reliabilityadjustment.

FIG. 6 is a signal flow diagram illustrating operations of a processorfor improving an image recognition rate according to variousembodiments.

The operations shown in FIG. 6 may be executed through a user interfacemodule 601, an image recognition module 602, and a user inputrecognition module 603. At least some of the operations performed by theuser interface module 601, the image recognition module 602, and theuser input recognition module 602 may be executed by the processor 120of FIG. 1. In addition, at least some of the operations performed by theuser interface module 601, the image recognition module 602, and theuser input recognition module 603 may be executed through at least oneof the processors 412, 430, 471 of FIG. 4.

According to various embodiments, at operation 610, the user interfacemodule 601 may acquire an image and transmit it to the image recognitionmodule 602. For example, this image may be a raw image acquired throughthe image sensor (e.g., 411 in FIG. 4) or a small raw image acquiredthrough the raw image processing module (e.g., 413 in FIG. 4). Inaddition, the image may be a decompressed version of an image fileacquired through the memory (e.g., 418 or 430 in FIG. 4).

According to various embodiments, at operation 615, the user interfacemodule 601 may acquire a user input and transmit it to the user inputrecognition module 603. This user input may be acquired from at leastone of, for example, the microphone, the keyboard, or thetouch-sensitive display. If the user input is an utterance acquired fromthe microcomputer, the acquired utterance may be converted into text andthen transmitted to the user input recognition module 603.Alternatively, the user input recognition module 603 may convert theutterance into text.

According to various embodiments, at operation 620, the imagerecognition module 602 (e.g., 473 of FIG. 4) may recognize one or moreobjects from the received image by using an algorithm obtained byapplying various learning engines (e.g., machine learning, deeplearning) to an image storage 604. According to one embodiment, theimage recognition module 602 may identify one or more image segmentsfrom the received image and recognize the identified image segment as atleast one object. In addition, the image recognition module 602 mayacquire reliability associated with object recognition by using theobtained algorithm.

According to various embodiments, at operation 625, the user inputrecognition module 603 (e.g., 472 of FIG. 4) may recognize, as anediting target, an object designated by the user from the received userinput.

According to various embodiments, at operation 630, the imagerecognition module 602 may determine whether there is a first priorityobject corresponding to the object (i.e., the editing target) recognizedby the user input recognition module 603 among one or more firstpriority objects recognized from the image.

According to various embodiments, if it is determined at operation 630that there is an object corresponding to the editing target among one ormore first priority objects, the image recognition module 602 mayincrease (i.e., adjust upward), at operation 635, the reliability of thefirst priority object corresponding to the editing target and also storea result of the reliability adjustment in the image storage 604 togetherwith the image. For example, a first image segment may be recognizedwith a probability (i.e., reliability) of ‘sea’ being 80% (firstpriority) and a probability of ‘sky’ being 20% (second priority). Also,a second image segment may be recognized with a probability of ‘sky’being 80% (first priority) and a probability of ‘sea’ being 20% (secondpriority). In this situation, when an object (i.e., a grammaticalobject) recognized from the user input is ‘sea’, a target that seems tobe designated by the user may be determined as the first image segmentbecause it is closer to the first image segment in terms of reliability(probabilistically) than to the second image segment. Therefore, thereliability of the first priority object ‘sea’ in the first imagesegment may be increased (i.e., adjusted upward). In addition, thereliability of the second priority object ‘sky’ in the first imagesegment may be decreased (i.e., adjusted downward).

According to various embodiments, if it is determined at operation 630that there is no object corresponding to the editing target among theone or more first priority objects, the image recognition module 602 maydetermine at operation 640 whether there is an object corresponding tothe editing target among one or more second priority objects recognizedfrom the image.

According to various embodiments, if it is determined at operation 640that there is an object corresponding to the editing target among one ormore second priority objects, the image recognition module 602 mayincrease (i.e., adjust upward), at operation 645, the reliability of thesecond priority object corresponding to the editing target, decrease(i.e., adjust downward) the reliability of the first priority object inthe corresponding image segment, and also store a result of thereliability adjustment in the image storage 604 together with the image.As in the above example, the first image segment may be recognized witha probability (i.e., reliability) of ‘sea’ being 80% (first priority)and a probability of ‘sky’ being 20% (second priority). Also, the secondimage segment may be recognized with a probability of ‘sky’ being 80%(first priority) and a probability of ‘sea’ being 20% (second priority).Further, a third image segment may be recognized with a probability of‘torch’ being 80% (first priority) and a probability of ‘sun’ being 20%(second priority). In this situation, when an object (i.e., agrammatical object) recognized from the user input is ‘sun’, thereliability of the second priority object ‘sun’ in the third imagesegment may be increased (i.e., adjusted upward), and the reliability ofthe first priority object ‘torch’ in the third image segment may bedecreased (i.e., adjusted downward) so that such priorities arereversed.

According to various embodiments, if it is determined at operation 640that there is no object corresponding to the editing target among one ormore second priority objects, the image recognition module 602 maytransmit, at operation 650, a message of requesting the user to directlydesignate the editing target to the user interface module 601. Forexample, if the image is divided into the first to fourth image segmentsas a result of image recognition, the fourth image segment may not beidentified as a corresponding object contrary to the other imagesegments being identified as one or more objects as described above. Inthis situation, if an object (a grammatical object) recognized from theuser input is a ‘cloud’, operation 650 may be performed.

According to various embodiments, in response to the request messagereceived from the image recognition module 602, the user interfacemodule 601 may output a message having the content of requesting adirect designation (e.g., “Touch the cloud.”) at operation 655. Forexample, the user interface module 601 may output a pop-up messagethrough the display or output a voice corresponding to the contentthrough the speaker.

According to various embodiments, at operation 660, the user interfacemodule 601 may acquire a user feedback (e.g., information about an imagesegment selected by the user) through the display or the input device(e.g., the microphone), and may transmit the user feedback to the imagerecognition module 602.

According to various embodiments, at operation 665, the imagerecognition module 602 may determine the reliability of the object,based on the user feedback, and may store result information associatedwith the determination of reliability in the image storage together withthe image. For example, if there is an object designated by the user ina list of previously recognized objects, the reliability of the objectmay be adjusted upward. If the object list does not have the objectdesignated by the user, the reliability of the object may be set to apredetermined value (e.g., 50%).

FIG. 7 is a diagram illustrating operations of an electronic device andan external electronic device according to various embodiments.

Referring to FIG. 7, an electronic device 710 may include all or a partof the electronic device 101 of FIG. 1 or of the electronic device 400of FIG. 4. An external electronic device 700 (e.g., 470 of FIG. 4) mayinclude an image recognition module 740, a user input recognition module720, and an image correction module 730. The image recognition module740, the user input recognition module 720, and the image correctionmodule 730 are functionally connected to each other, thereby freelytransferring and sharing data therebetween. According to one embodiment,the image recognition module 740, the user input recognition module 720,and the image correction module 730 may be logical modules, and thus atleast one processor (e.g., 412, 430, 471 in FIG. 4) may perform theiroperations. According to one embodiment, at least one of the imagerecognition module 740, the user input recognition module 720, and theimage correction module 730 may be implemented as hardware within theprocessor (e.g., 412, 430, or 471 in FIG. 4). According to oneembodiment, the processor (e.g., 412 or 430 in FIG. 4) of the electronicdevice 710 may perform the same operation as that of the imagecorrection module 730.

According to various embodiments, an image 751 acquired by theelectronic device 710 may be transmitted to the image recognition module740 of the external electronic device 700. In addition, a user input 753(e.g., utterance or text) acquired by the electronic device 710 may betransmitted to the user input recognition module 720 of the externalelectronic device 710.

According to various embodiments, the user input recognition module 720(e.g., 472 of FIG. 4) may perform object recognition from the user inputand then transmit a user input recognition result 754 to the electronicdevice 710 or the image correction module 730.

According to various embodiments, the image correction module 730 mayinclude various members for image correction, such as the ISP (e.g., 476in FIG. 4), the encoder (e.g., 475 in FIG. 4), and the preprocessingmodule (e.g., 474 in FIG. 4). The image correction module 730 mayperform image correction based on the image recognition result 752 andthe user input recognition result 754. Then, the image correction module730 may transmit a corrected image 756 to the electronic device 710.

According to various embodiments, the image recognition module 740(e.g., 473 in FIG. 4) may perform object recognition from an image.According to one embodiment, when performing the object recognitionbased on a deep learning recognition model, the image recognition module740 may use differently learned recognition models for the respectiveusers. For example, personalized recognition models 750 in which auser's native language, user-related persons, a residential environment,etc. are considered may be stored in the database, and the imagerecognition module 740 may retrieve a suitable recognition model for theuser of the electronic device 710 to perform the object recognition. Theimage recognition module 740 may transmit the image recognition result752 to the electronic device 710 or the image correction module 730.

According to various embodiments, the image recognition module 740 mayadjust the object reliability, based on a feedback 757 received from theelectronic device 710 as a user's response to the corrected image 756,and may also perform an update 758 of the user's recognition model,based on result information related to the reliability adjustment.Various examples regarding the user feedback and the reliabilityadjustment are as previously discussed with reference to FIGS. 4 to 6.

According to various embodiments of the present disclosure, anelectronic device may comprise a communication module, and a processorfunctionally connected to the communication module. The processor may beconfigured to identify an image including one or more objects, toacquire one or more first objects corresponding to at least a part ofthe one or more objects and recognized at least based on an imagerecognition scheme, and one or more reliabilities associated withrecognition of the one or more first objects, to acquire an inputincluding information having one or more words associated with theimage, to acquire one or more second objects corresponding to at least apart of the one or more words and recognized at least based on acharacter recognition scheme, to, when there is at least one objectcorresponding to the one or more second objects among the one or morefirst objects, adjust at least one reliability corresponding to the atleast one object among the one or more reliabilities, and to recognizethe one or more first objects by using the image recognition scheme atleast based on the one or more reliabilities including the adjusted atleast one reliability.

The processor may be further configured to adjust upward at least onereliability corresponding to the at least one object among the one ormore reliabilities when there is the at least one object correspondingto the one or more second objects among the one or more first objects.

The processor may be further configured to determine, among the firstobjects, a first object set corresponding to a first priority and asecond object set corresponding to a second priority lower than thefirst priority, to, when an object corresponding to the one or moresecond objects is included in the first object set, adjust upward areliability of the object corresponding to the one or more secondobjects included in the first object set, and to, when an objectcorresponding to the one or more second objects is not included in thefirst object set and included in the second object set, adjust upward areliability of the object corresponding to the one or more secondobjects included in the second object set and adjust downward areliability of at least one object included in the first object set.

The processor may be further configured to output an input requestcorresponding to the one or more second objects when there is no objectcorresponding to the one or more second objects among the one or morefirst objects.

The processor may be further configured to select one of a plurality ofimage segments of the image, based on a user reaction to the inputrequest, to determine the one or more second objects as an objectcorresponding to the selected image segment, and to determine areliability of the object corresponding to the selected image segment.

The processor may be further configured to adjust upward the reliabilityof the object corresponding to the selected image segment when there isthe object corresponding to the selected image segment in the one ormore first objects.

The processor may be further configured to display a corrected imagewhich is corrected from the image, at least based on the imagerecognition scheme, to acquire another input associated with thedisplayed corrected image, and to adjust at least one reliability amongthe one or more reliabilities, based on the another input according tothe character recognition scheme.

The processor may be further configured to, when the input includes auser utterance, convert the utterance into text as a part of thecharacter recognition scheme, and to recognize the one or more wordsfrom the text.

The electronic device may further comprise a camera functionallyconnected to the processor, and the processor may be further configuredto acquire a first image by using the camera, to generate a second imagehaving a smaller size than a size of the first image by using the firstimage, and to transmit the second image to the external electronicdevice through the communication module such that the externalelectronic device recognizes the one or more first objects and the oneor more reliabilities from the second image.

The electronic device may further comprise a microphone functionallyconnected to the processor, and the processor may be further configuredto acquire a user input from the microphone, and to transmit the userinput to the external electronic device through the communication modulesuch that the external electronic device recognizes the one or moresecond objects from the user input.

The processor may be further configured to store the adjustedreliability together with the image in an image storage which is used inobject recognition based on the image recognition scheme.

The electronic device may further comprise a camera functionallyconnected to the processor, and the processor may be further configuredto acquire an image by using the camera, and to recognize the one ormore first objects and the one or more reliabilities by using theacquired image.

According to various embodiments of the present disclosure, anelectronic device may comprise a touch-sensitive display, an inputdevice, a camera, a communication module, and a processor functionallyconnected to the touch-sensitive display, the input device, the camera,and the communication module. The processor may be configured to acquirea first image through the camera, to display a second imagecorresponding to the first image on the touch-sensitive display, toacquire a user input through at least one of the touch-sensitive displayor the input device while the second image is displayed, to transmit thefirst image and the user input to the external electronic device throughthe communication module such that the external electronic deviceperforms recognition of an object corresponding to the user input amongobjects of the first image, to receive a result of the recognition fromthe external electronic device, to acquire a first user reaction to theresult through at least one of the touch-sensitive display or the inputdevice, and to transmit the first reaction to the external electronicdevice through the communication module such that the externalelectronic device adjusts a reliability associated with the objectrecognition, based on the first reaction.

The processor may be further configured to receive, from the externalelectronic device through the communication module, a message indicatingno object corresponding to the user input among objects of the firstimage or indicating a re-request of a user input, to acquire a seconduser reaction to the message through at least one of the touch-sensitivedisplay or the input device, and to transmit the second reaction to theexternal electronic device through the communication module such thatthe external electronic device adjusts the reliability associated withthe object recognition, based on the second reaction.

According to various embodiments of the present disclosure, a method ofoperating an electronic device may comprise identifying an imageincluding one or more objects; acquiring one or more first objectscorresponding to at least a part of the one or more objects andrecognized at least based on an image recognition scheme, and one ormore reliabilities associated with recognition of the one or more firstobjects; acquiring an input including information having one or morewords associated with the image; acquiring one or more second objectscorresponding to at least a part of the one or more words and recognizedat least based on a character recognition scheme; when there is at leastone object corresponding to the one or more second objects among the oneor more first objects, adjust at least one reliability corresponding tothe at least one object among the one or more reliabilities; andrecognize the one or more first objects by using the image recognitionscheme at least based on the one or more reliabilities including theadjusted at least one reliability.

While the present disclosure has been particularly shown and describedwith reference to exemplary embodiments thereof, it is clearlyunderstood that the same is by way of illustration and example only andis not to be taken in conjunction with the present disclosure. It willbe understood by those skilled in the art that various changes in formand details may be made therein without departing from the subjectmatter and scope of the present disclosure.

What is claimed is:
 1. An electronic device comprising: a processorconfigured to: receive an image including one or more objects, acquire afirst one or more of the received one or more objects, acquire one ormore reliability measures associated with the acquired one or more firstobjects, receive an input including information having one or morewords, acquire one or more second objects corresponding to at least apart of the one or more words, determine, among the first objects, afirst object set corresponding to a first priority and a second objectset corresponding to a second priority lower than the first priority,when there is at least one object corresponding to the one or moresecond objects among the one or more first objects, adjust at least onereliability measure corresponding to the at least one object among theacquired one or more reliability measures, including: when an objectcorresponding to the one or more second objects is included in the firstobject set, adjust upward a reliability of the object corresponding tothe one or more second objects included in the first object set, andrecognize the one or more first objects by using an image recognitionscheme at least based on the one or more reliabilities including theadjusted at least one reliability.
 2. The electronic device of claim 1,wherein the processor is further configured to adjust upward the atleast one reliability measure.
 3. The electronic device of claim 1,wherein adjusting the at least one reliability measure furthercomprises: when the object corresponding to the one or more secondobjects is not included in the first object set and included in thesecond object set, adjust upward a reliability of the objectcorresponding to the one or more second objects included in the secondobject set and adjust downward a reliability of at least one objectincluded in the first object set.
 4. The electronic device of claim 1,wherein the processor is further configured to output an input requestcorresponding to the one or more second objects when there are no objectcorresponding to the one or more second objects among the one or morefirst objects.
 5. The electronic device of claim 4, wherein theprocessor is further configured to: select one of a plurality of imagesegments of the image, based on a user reaction to the input request,determine the one or more second objects as an object corresponding tothe selected image segment, and determine a reliability of the objectcorresponding to the selected image segment.
 6. The electronic device ofclaim 5, wherein the processor is further configured to adjust upwardthe reliability of the object corresponding to the selected imagesegment when there is the object corresponding to the selected imagesegment in the one or more first objects.
 7. The electronic device ofclaim 1, wherein the processor is further configured to: display acorrected image which is corrected from the image, at least based on theimage recognition scheme, acquire another input associated with thedisplayed corrected image, and adjust at least one reliability among theone or more reliabilities, based on the another input.
 8. The electronicdevice of claim 1, wherein the processor is further configured to, whenthe input includes a user utterance, convert the utterance into text,and to recognize the one or more words from the text.
 9. The electronicdevice of claim 1, further comprising: a camera functionally connectedto the processor, wherein the processor is further configured to:acquire a first image by using the camera, generate a second imagehaving a smaller size than a size of the first image by using the firstimage, and transmit the second image to an external electronic devicethrough a communication module such that the external electronic devicerecognizes the one or more first objects and the one or morereliabilities from the second image.
 10. The electronic device of claim1, further comprising: a microphone functionally connected to theprocessor, wherein the processor is further configured to: acquire auser input from the microphone, and transmit the user input to anexternal electronic device through a communication module such that aexternal electronic device recognizes the one or more second objectsfrom the user input.
 11. The electronic device of claim 1, wherein theprocessor is further configured to store an adjusted reliabilitytogether with the image in an image storage which is used in objectrecognition based on the image recognition scheme.
 12. The electronicdevice of claim 1, further comprising: a camera functionally connectedto the processor, wherein the processor is further configured to:acquire the image by using the camera, and recognize the one or morefirst objects and the one or more reliabilities by using the acquiredimage.
 13. An electronic device comprising: a touch-sensitive display;an input device; a camera; a communication module; and a processorfunctionally connected to the touch-sensitive display, the input device,the camera, and the communication module, wherein the processor isconfigured to: acquire a first image through the camera, display asecond image corresponding to the first image on the touch-sensitivedisplay, acquire a user input through at least one of thetouch-sensitive display or the input device while the second image isdisplayed, transmit the first image and the user input to an externalelectronic device through the communication module such that theexternal electronic device performs recognition of an objectcorresponding to the user input among objects of the first image,including determining, among the first objects, a first object setcorresponding to a first priority and a second object set correspondingto a second priority lower than the first priority, receive a result ofthe recognition from the external electronic device, acquire a firstuser input responsive to the result through at least one of thetouch-sensitive display or the input device, and transmit the first userinput to the external electronic device through the communication modulesuch that the external electronic device adjusts a reliabilityassociated with the object recognition, based on the first user input,including: when an object corresponding to the one or more secondobjects is included in the first object set, adjusting upward areliability of the object corresponding to the one or more secondobjects included in the first object set.
 14. The electronic device ofclaim 13, wherein the processor is further configured to: receive, fromthe external electronic device through the communication module, amessage indicating no object corresponding to the user input amongobjects of the first image or indicating a re-request of a user input,acquire a second user input responsive to the message through at leastone of the touch-sensitive display or the input device, and transmit thesecond user input to the external electronic device through thecommunication module such that the external electronic device adjuststhe reliability associated with the object recognition, based on thesecond user input.
 15. A method of operating an electronic device, themethod comprising: identifying an image including one or more objects;acquiring one or more first objects corresponding to at least a part ofthe one or more objects, and one or more reliabilities associated withrecognition of the one or more first objects; acquiring an inputincluding information having one or more words associated with theimage; acquiring one or more second objects corresponding to at least apart of the one or more words; determining, among the first objects, afirst object set corresponding to a first priority and a second objectset corresponding to a second priority lower than the first priority;when there is at least one object corresponding to the one or moresecond objects among the one or more first objects, adjust at least onereliability corresponding to the at least one object among the one ormore reliabilities, including: when an object corresponding to the oneor more second objects is included in the first object set, adjustingupward a reliability of the object corresponding to the one or moresecond objects included in the first object set; and recognize the oneor more first objects by using an image recognition scheme at leastbased on the one or more reliabilities including the adjusted at leastone reliability.